Graphical user interface for displaying automatically segmented individual parts of anatomy in a surgical navigation system

ABSTRACT

A surgical navigation system includes a source of a patient anatomy data, wherein the patient anatomy data comprises a three-dimensional reconstruction of a segmented model comprising at least two sections representing parts of the anatomy. A surgical navigation image generator is configured to generate a surgical navigation image comprising the patient anatomy. A 3D display system is configured to show the surgical navigation image wherein the display of the patient anatomy is selectively configurable such that at least one section of the anatomy is displayed and at least one other section of the anatomy is not displayed.

TECHNICAL FIELD

The present disclosure relates to graphical user interfaces for surgicalnavigation systems, in particular to a system and method for operativeplanning and real time execution of a surgical procedure includingdisplaying automatically segmented individual parts of the patientanatomy.

BACKGROUND

Some of typical functions of a computer-assisted surgery (CAS) systemwith navigation include presurgical planning of a procedure andpresenting preoperative diagnostic information and images in usefulformats. The CAS system presents status information about a procedure asit takes place in real time, displaying the preoperative plan along withintraoperative data. The CAS system may be used for procedures intraditional operating rooms, interventional radiology suites, mobileoperating rooms or outpatient clinics. The procedure may be any medicalprocedure, whether surgical or non-surgical.

Surgical navigation systems are used to display the position andorientation of surgical instruments and medical implants with respect topresurgical or intraoperative medical imagery datasets of a patient.These images include pre and intraoperative images, such astwo-dimensional (2D) fluoroscopic images and three-dimensional (3D)magnetic resonance imaging (MRI) or computed tomography (CT).

Navigation systems locate markers attached or fixed to an object, suchas surgical instruments and patient. Most commonly these trackingsystems are optical and electro-magnetic. Optical tracking systems haveone or more stationary cameras that observe passive reflective markersor active infrared LEDs attached to the tracked instruments or thepatient. Eye-tracking solutions are specialized optical tracking systemsthat measure gaze and eye motion relative to a user's head.Electro-magnetic systems have a stationary field generator that emits anelectromagnetic field that is sensed by coils integrated into trackedmedical tools and surgical instruments.

SUMMARY OF THE INVENTION

Incorporating image segmentation processes that automatically identifyvarious bone landmarks, based on their density, can increase planningaccuracy. One such bone landmark is the spinal pedicle, which is made upof dense cortical bone making its identification utilizing imagesegmentation easier. The pedicle is used as an anchor point for varioustypes of medical implants. Achieving proper implant placement in thepedicle is heavily dependent on the trajectory selected for implantplacement. Ideal trajectory is identified by surgeon based on review ofadvanced imaging (e.g., CT or MRI), goals of the surgical procedure,bone density, presence or absence of deformity, anomaly, prior surgery,and other factors. The surgeon then selects the appropriate trajectoryfor each spinal level. Proper trajectory generally involves placing anappropriately sized implant in the center of a pedicle. Idealtrajectories are also critical for placement of inter-vertebralbiomechanical devices.

Another example is placement of electrodes in the thalamus for thetreatment of functional disorders, such as Parkinson's. The mostimportant determinant of success in patients undergoing deep brainstimulation surgery is the optimal placement of the electrode. Propertrajectory is defined based on preoperative imaging (such as MRI or CT)and allows for proper electrode positioning.

Another example is minimally invasive replacement of prosthetic/biologicmitral valve in for the treatment of mitral valve disorders, such asmitral valve stenosis or regurgitation. The most important determinantof success in patients undergoing minimally invasive mitral valvesurgery is the optimal placement of the three dimensional valve.

The fundamental limitation of surgical navigation systems is that theyprovide restricted means of communicating to the surgeon.Currently-available navigation systems present some drawbacks.

Typically, one or several computer monitors are placed at some distanceaway from the surgical field. They require the surgeon to focus thevisual attention away from the surgical field to see the monitors acrossthe operating room. This results in a disruption of surgical workflow.Moreover, the monitors of current navigation systems are limited todisplaying multiple slices through three-dimensional diagnostic imagedatasets, which are difficult to interpret for complex 3D anatomy.

The fact that the screen of the surgical navigation system is locatedaway from the region of interest (ROI) of the surgical field requiresthe surgeon to continuously look back and forth between the screen andthe ROI. This task is not intuitive and results in a disruption tosurgical workflow and decreases planning accuracy.

When defining and later executing an operative plan, the surgeoninteracts with the navigation system via a keyboard and mouse,touchscreen, voice commands, control pendant, foot pedals, hapticdevices, and tracked surgical instruments. Based on the complexity ofthe 3D anatomy, it can be difficult to simultaneously position andorient the instrument in the 3D surgical field only based on theinformation displayed on the monitors of the navigation system.Similarly, when aligning a tracked instrument with an operative plan, itis difficult to control the 3D position and orientation of theinstrument with respect to the patient anatomy. This can result in anunacceptable degree of error in the preoperative plan that willtranslate to poor surgical outcome.

One aspect of the invention is a surgical navigation system comprising:a source of a patient anatomy data; wherein the patient anatomy datacomprises a three-dimensional reconstruction of a segmented modelcomprising at least two sections representing parts of the anatomy; asurgical navigation image generator configured to generate a surgicalnavigation image comprising the patient anatomy; a 3D display systemconfigured to show the surgical navigation image wherein the display ofthe patient anatomy is selectively configurable such that at least onesection of the anatomy is displayed and at least one other section ofthe anatomy is not displayed.

The system may further comprise a tracking system for real-time trackingof: a surgeon's head, a see-through visor of the 3D display system and apatient anatomy to provide current position and/or orientation data;wherein the surgical navigation image generator is configured togenerate the surgical navigation image in accordance to the currentposition and/or orientation data provided by the tracking system.

The system may further comprise a source of at least one of: anoperative plan and a virtual surgical instrument model; wherein thetracking system is further configured for real-time tracking of surgicalinstruments; wherein the surgical navigation image further comprises athree-dimensional image representing a virtual image of the surgicalinstruments.

The virtual image of the surgical instruments can be configured toindicate the suggested positions and/or orientations of the surgicalinstruments according to the operative plan data.

The three-dimensional image of the surgical navigation image may furthercomprise a graphical cue indicating the required change of positionand/or orientation of the surgical instrument to match the suggestedposition and/or orientation according to the pre-operative plan data.

The surgical navigation image may further comprise a set of orthogonal(axial, sagittal, and coronal) and/or arbitrary planes of the patientanatomy data.

The 3D display system may comprise a 3D projector for projecting thesurgical navigation image onto a see-through projection screen, which ispartially transparent and partially reflective, for showing the surgicalnavigation image.

The 3D display system may comprise a 3D projector for projecting thesurgical navigation image onto an opaque projection screen for showingthe surgical navigation image for emission towards the see-throughmirror, which is partially transparent and partially reflective.

The 3D display may comprise a 3D projector for projecting the surgicalnavigation image towards a plurality of opaque mirrors for reflectingthe surgical navigation image towards an opaque projection screen forshowing the surgical navigation image for emission towards thesee-through mirror, which is partially transparent and partiallyreflective.

The 3D display may comprise a 3D monitor for showing the surgicalnavigation image for emission towards the see-through mirror which ispartially transparent and partially reflective.

The 3D display may comprise a see-through 3D screen, which is partiallytransparent and partially emissive, for showing the surgical navigationimage.

The see-through visor can be configured to be positioned, when thesystem is in use, at a distance from the surgeon's head which is shorterthan the distance from the surgical field of the patient anatomy.

The surgical navigation image generator can be controllable by an inputinterface comprising at least one of: foot-operable pedals, amicrophone, a joystick, an eye-tracker.

The tracking system may comprise plurality of arranged fiducial markers,including a head array, a display array, a patient anatomy array, aninstrument array; and a fiducial marker tracker configured to determinein real time the positions and orientations of each of the components ofthe surgical navigation system.

At least one of the head array, the display array, the patient anatomyarray, the instrument array may contain several fiducial markers thatare not all coplanar.

The patient anatomy data may comprise output data of a semanticsegmentation process of an anatomy scan image.

The system may further comprise a convolutional neural network systemconfigured to perform the semantic segmentation process to generate thepatient anatomy data.

The convolutional neural network (CNN) system may comprise: at least onenon-transitory processor-readable storage medium that stores at leastone of processor-executable instructions or data; and at least oneprocessor communicably coupled to at least one non-transitoryprocessor-readable storage medium, wherein that at least one processor:receives segmentation learning data comprising a plurality of batches oflabeled anatomical image sets, each image set comprising image datarepresentative of a series of slices of a three-dimensional bonystructure of the anatomy, and each image set including at least onelabel which identifies the region of a particular part of the bonystructure depicted in each image of the image set, wherein the labelindicates one of a plurality of classes indicating parts of the boneanatomy; trains a segmentation CNN, that is a fully convolutional neuralnetwork model with layer skip connections to segment semantically atleast one part of the bony structure utilizing the received segmentationlearning data; and stores the trained segmentation CNN in at least onenon-transitory processor-readable storage medium of the machine learningsystem.

Training the CNN model may include training a CNN model including acontracting path and an expanding path. The contracting path may includea number of convolutional layers, a number of pooling layers and dropoutlayers. Each pooling and dropout layer may be preceded by at least oneconvolutional layer. The expanding path may include a number ofconvolutional layers, a number of upsampling layers and a concatenationof feature maps from previous layers. Each upsampling layer may bepreceded by at least one convolutional layer and may include a transposeconvolution operation which performs upsampling and interpolation with alearned kernel.

Training a CNN model may include training a CNN model to segment atleast one part of the anatomical structure utilizing the receivedlearning data and, subsequent to each upsampling layer, the CNN modelmay include a concatenation of feature maps from a corresponding layerin the contracting path through a skip connection. Receiving learningdata may include receiving preoperative or intraoperative images of thebony structure. Training a CNN model may include training a CNN model tosegment at least one part of the anatomical structure utilizing thereceived learning data, and the CNN model may include a contracting pathwhich may include a first convolutional layer, which may have between 1and 256 feature maps. Training a CNN model may include training a CNNmodel which may include a plurality of convolutional layers to segmentat least one part of the anatomical structure of the vertebrae utilizingthe received learning data, and each convolutional layer may include aconvolutional kernel of sizes 2n+1×2n+1, with n being a natural number,and a selectable stride. Training a CNN model may include training a CNNmodel which may include a plurality of pooling layers to segment atleast one part of the anatomical structure utilizing the receivedlearning data, and each pooling layer may include an n×n maximum orother type of pooling, with a selectable stride, with n being a naturalnumber.

A CNN model may include training a CNN model to segment at least onepart of the anatomical structure utilizing the received learning data,and the CNN model may include a plurality of pooling layers and aplurality of upsampling layers.

A CNN model may include training a CNN model which may include aplurality of convolutional layers to segment at least one part of theanatomical structure utilizing the received learning data, and the CNNmodel may pad the input to each convolutional layer using a zero paddingoperation.

A CNN model may include training a CNN model to segment at least onepart of the anatomical structure utilizing the received learning data,and the CNN model may include a plurality of nonlinear activationfunction layers. The method may further include augmenting, by at leastone processor, the learning data via modification of at least some ofthe image data in the plurality of batches of labeled image sets.

The method may further include modifying, by at least one processor, atleast some of the image data in the plurality of batches of labeledimage sets according to at least one of: a horizontal flip, a verticalflip, a shear amount, a shift amount, a zoom amount, a rotation amount,a brightness level, or a contrast level, additive noise of Gaussianand/or Poisson distribution and Gaussian blur.

The CNN model may include a plurality of hyperparameters stored in atleast one non-transitory processor-readable storage medium, and mayfurther include configuring, by at least one processor, the CNN modelaccording to a plurality of configurations; for each of the plurality ofconfigurations, validating, by at least one processor, the accuracy ofthe CNN model; and selecting, by at least one processor, at least oneconfiguration based at least in part on the accuracies determined by thevalidations.

The method may further include for each image set, identifying, by atleast one processor, whether the image set is missing a label for any ofa plurality of parts of the anatomical structure; and for image setsidentified as missing at least one label, modifying, by at least oneprocessor, a training loss function to account for the identifiedmissing labels. Receiving learning data may include receiving image datawhich may include volumetric images, and each label may include avolumetric label mask or contour.

A CNN model may include training a CNN model which may include aplurality of convolutional layers to segment at least one part of theanatomical structure utilizing the received learning data, and eachconvolutional layer of the CNN model may include a convolutional kernelof size N×N×K pixels, where N and K are positive integers.

A CNN model may include training a CNN model which may include aplurality of convolutional layers to segment at least one part of theanatomical structure utilizing the received learning data, and eachconvolutional layer of the CNN model may include a convolutional kernelof size N×M pixels, where N and M are positive integers. Receivinglearning data may include receiving image data representative of labeledanatomical parts. Training a CNN model may include training a CNN modelto segment at least one part of the anatomical structure utilizing thereceived learning data, and for each processed image, the CNN model mayutilize data for at least one image which is at least one of: adjacentto the processed image with respect to space

A method of operating a machine learning system may include at least onenon-transitory processor-readable storage medium that stores at leastone of processor-executable instructions or data, and at least oneprocessor communicably coupled to at least one non-transitoryprocessor-readable storage medium. The method may be summarized asincluding receiving, by at least one processor, image data whichrepresents an anatomical structure; processing, by at least oneprocessor, the received image data through a fully convolutional neuralnetwork (CNN) model to generate per-class probabilities for each pixelof each image of the image data, each class corresponding to one of aplurality of parts of the anatomical structure represented by the imagedata; and for each image of the image data, generating, by at least oneprocessor, a probability map for each of the plurality of classes usingthe generated per-class probabilities; and storing, by at least oneprocessor, the generated probability maps in at least one non-transitoryprocessor-readable storage medium.

Processing the received image data through the CNN model may includeprocessing the received image data through a CNN model which may includea contracting path and an expanding path. The contracting path mayinclude a number of convolutional layers and a number of pooling layers,each pooling layer preceded by at least one convolutional layer. Theexpanding path may include a number of convolutional layers and a numberof upsampling layers, each upsampling layer preceded by at least oneconvolutional layer, and may include a transpose convolution operationwhich performs upsampling and interpolation with a learned kernel.Receiving image data may include, for example, receiving image data thatis representative of a vertebrae in a spine. The method may furtherinclude autonomously causing, by the at least one processor, anindication of at least one of the plurality of parts of the anatomicalstructure to be displayed on a display based at least in part on thegenerated probability maps.

The method may further include post-processing, by at least oneprocessor, the processed image data to ensure at least one physicalconstraint is met. Receiving image data may include, for example,receiving image data that may be representative of vertebrae, and atleast one physical constraint may include at least one of: constraintson the volumes of anatomical parts of the bony structure, such as aspine, coincidence and connections of the anatomical parts of thevertebrae, such as the vertebral body must be connected to two pedicles,spinous process must be connected to the lamina and cannot be connectedto the vertebral body etc.

The method may further include for each image of the image data,transforming, by at least one processor, the plurality of probabilitymaps into a label mask by setting the class of each pixel to the classwith the highest probability.

The method may further include for each image of the image data,setting, by at least one processor, the class of each pixel to abackground class when all of the class probabilities for the pixel arebelow a determined threshold.

The method may further include for each image of the image data,setting, by at least one processor, the class of each pixel to abackground class when the pixel is not part of a largest connectedregion for the class to which the pixel is associated.

The method may further include converting, by at least one processor,each of the label masks for the image data combined into a 3D volume andfurther converting it into an alternative representation in the form ofa polygonal mesh.

The method may further include autonomously causing, by at least oneprocessor, the generated mesh to be displayed with the image data on adisplay.

The method may further include receiving, by at least one processor, auser modification of at least one of the displayed volumes and/or meshesin terms of change of color, opacity, changing the mesh decimation; andstoring, by at least one processor, the modified volumes and/or meshesin at least one non-transitory processor-readable storage medium. Themethod may further include determining, by at least one processor, thevolume of at least one of the plurality of parts of the anatomicalstructure utilizing the generated volume or mesh.

The method may further include causing, by at least one processor, thedetermined volume of at least one of the plurality of parts of theanatomical structure to be displayed on a display. Receiving image datamay include receiving volumetric image data or polygonal mesh data.Processing the received image data through a CNN model may includeprocessing the received image data through a CNN model in which eachconvolutional layer may include a convolutional kernel of sizes N×N×Kpixels, where N and K are positive integers.

Another aspect of the invention is a method for providing an augmentedreality image during an operation, comprising: providing a source of apatient anatomy data; wherein the patient anatomy data comprises athree-dimensional reconstruction of a segmented model comprising atleast two sections representing parts of the anatomy; generating, by asurgical navigation image generator, a surgical navigation imagecomprising the patient anatomy; showing the surgical navigation image at3D display system and configuring the display of the patient anatomysuch that at least one section of the anatomy is displayed and at leastone other section of the anatomy is not displayed.

These and other features, aspects and advantages of the invention willbecome better understood with reference to the following drawings,descriptions and claims.

BRIEF DESCRIPTION OF FIGURES

Various embodiments are herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIG. 1A shows a layout of a surgical room employing the surgicalnavigation system in accordance with an embodiment of the invention;

FIG. 1B shows a layout of a surgical room employing the surgicalnavigation system in accordance with an embodiment of the invention;

FIG. 1C shows a layout of a surgical room employing the surgicalnavigation system in accordance with an embodiment of the invention;

FIG. 2A shows the connections between the different components thatinteract in accordance with an embodiment of the invention;

FIG. 2B shows components of the surgical navigation system in accordancewith an embodiment of the invention;

FIG. 3A shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3B shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3C shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3D shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3E shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3F shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3G shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3H shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3I shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 4A shows a different embodiment of a 3D display system;

FIG. 4B shows another embodiment of a 3D display system;

FIG. 4C shows another embodiment of a 3D display system;

FIG. 4D shows another embodiment of a 3D display system;

FIG. 4E shows another embodiment of a 3D display system;

FIG. 5A shows eye tracking in accordance with an embodiment of theinvention;

FIG. 5B shows eye tracking in accordance with an embodiment of theinvention;

FIG. 6 shows a 3D representation of the results of the semanticsegmentation on one vertebrae in accordance with an embodiment of theinvention;

FIG. 7A shows an example of a CT image of a spine;

FIG. 7B shows another example of a CT image of a spine;

FIG. 7C shows another example of a CT image of a spine;

FIG. 7D shows another example of a CT image of a spine;

FIG. 7E shows another example of a CT image of a spine;

FIG. 7F shows a semantic segmented image corresponding to the CT imageof FIG. 7A, in accordance with an embodiment of the invention;

FIG. 7G shows a semantic segmented image corresponding to the CT imageof FIG. 7B, in accordance with an embodiment of the invention;

FIG. 7H shows a semantic segmented image corresponding to the CT imageof FIG. 7C, in accordance with an embodiment of the invention;

FIG. 7I shows a semantic segmented image corresponding to the CT imageof FIG. 7D, in accordance with an embodiment of the invention;

FIG. 7J shows a semantic segmented image corresponding to the CT imageof FIG. 7E, in accordance with an embodiment of the invention;

FIG. 8A shows an enlarged view of a LDCT scan;

FIG. 8B shows an enlarged view of a HDCT scan;

FIG. 8C shows a low power magnetic resonance scan of a neck portion;

FIG. 8D shows a higher power magnetic resonance scan of the same neckportion as FIG. 8C;

FIG. 9 shows a denoising CNN architecture in accordance with anembodiment of the invention;

FIG. 10 shows a segmentation CNN architecture in accordance with anembodiment of the invention;

FIG. 11 shows a flowchart of a training process in accordance with anembodiment of the invention;

FIG. 12 shows a flowchart of an inference process for the denoising CNNin accordance with an embodiment of the invention;

FIG. 13 shows a flowchart of an inference process for the segmentationCNN in accordance with an embodiment of the invention;

FIG. 14A shows a sample image of a CT spine scan;

FIG. 14B shows a sample image of the segmentation of the sample image ofFIG. 14A in accordance with an embodiment of the invention;

FIG. 15 shows a schematic of a system for implementing the segmentationCNN in accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description is of the best currently contemplatedmodes of carrying out the invention. The description is not to be takenin a limiting sense, but is made merely for the purpose of illustratingthe general principles of the invention.

The system presented herein, in accordance with one embodiment,comprises a 3D display system 140 to be implemented directly on realsurgical applications in a surgical room as shown in FIGS. 1A-1C. The 3Ddisplay system 140 as shown in the embodiment of FIGS. 1A-1C comprises a3D display 142 for emitting a surgical navigation image 142A towards asee-through mirror 141 that is partially transparent and partiallyreflective, such that an augmented reality image 141A collocated withthe patient anatomy in the surgical field 108 underneath the see-throughmirror 141 is visible to a viewer looking from above the see-throughmirror 141 towards the surgical field 108.

The surgical room typically comprises a floor 101 on which an operatingtable 104 is positioned. A patient 105 lies on the operating table 104while being operated by a surgeon 106 with the use of various surgicalinstruments 107. The surgical navigation system as described in detailsbelow can have its components, in particular the 3D display system 140,mounted to a ceiling 102, or alternatively to the floor 101 or a sidewall 103 of the operating room. Furthermore, the components, inparticular the 3D display system 140, can be mounted to an adjustableand/or movable floor-supported structure (such as a tripod). Componentsother than the 3D display system 140, such as the surgical imagegenerator 131, can be implemented in a dedicated computing device 109,such as a stand-alone PC computer, which may have its own inputcontrollers and display(s) 110.

In general, the system is designed for use in such a configurationwherein the distance d1 between the surgeon's eyes and the see-throughmirror 141, is shorter than the distance d2, between the see-throughmirror 141 and the operative field at the patient anatomy 105 beingoperated.

FIG. 2A shows a functional schematic presenting connections between thecomponents of the surgical navigation system and FIG. 2B shows examplesof physical embodiments of various components.

The surgical navigation system comprises a tracking system for trackingin real time the position and/or orientation of various entities toprovide current position and/or orientation data. For example, thesystem may comprise a plurality of arranged fiducial markers, which aretrackable by a fiducial marker tracker 125. Any known type of trackingsystem can be used, for example in case of a marker tracking system,4-point marker arrays are tracked by a three-camera sensor to providemovement along six degrees of freedom. A head position marker array 121can be attached to the surgeon's head for tracking of the position andorientation of the surgeon and the direction of gaze of the surgeon—forexample, the head position marker array 121 can be integrated with thewearable 3D glasses 151 or can be attached to a strip worn oversurgeon's head.

A display marker array 122 can be attached to the see-through mirror 141of the 3D display system 140 for tracking its position and orientation,as the see-through mirror 141 is movable and can be placed according tothe current needs of the operative setup.

A patient anatomy marker array 123 can be attached at a particularposition and orientation of the anatomy of the patient.

A surgical instrument marker array 124 can be attached to the instrumentwhose position and orientation shall be tracked.

Preferably, the markers in at least one of the marker arrays 121-124 arenot coplanar, which helps to improve the accuracy of the trackingsystem.

Therefore, the tracking system comprises means for real-time tracking ofthe position and orientation of at least one of: a surgeon's head 106, a3D display 142, a patient anatomy 105, and surgical instruments 107.Preferably, all of these elements are tracked by a fiducial markertracker 125.

A surgical navigation image generator 131 is configured to generate animage to be viewed via the see-through mirror 141 of the 3D displaysystem. It generates a surgical navigation image 142A comprising data ofat least one of: the pre-operative plan 161 (which are generated andstored in a database before the operation), data of the intra-operativeplan 162 (which can be generated live during the operation), data of thepatient anatomy scan 163 (which can be generated before the operation orlive during the operation) and virtual images 164 of surgicalinstruments used during the operation (which are stored as 3D models ina database).

The surgical navigation image generator 131, as well as other componentsof the system, can be controlled by a user (i.e. a surgeon or supportstaff) by one or more user interfaces 132, such as foot-operable pedals(which are convenient to be operated by the surgeon), a keyboard, amouse, a joystick, a button, a switch, an audio interface (such as amicrophone), a gesture interface, a gaze detecting interface etc. Theinput interface(s) are for inputting instructions and/or commands.

All system components are controlled by one or more computer which iscontrolled by an operating system and one or more software applications.The computer may be equipped with a suitable memory which may storecomputer program or programs executed by the computer in order toexecute steps of the methods utilized in the system. Computer programsare preferably stored on a non-transitory medium. An example of anon-transitory medium is a non-volatile memory, for example a flashmemory while an example of a volatile memory is RAM. The computerinstructions are executed by a processor. These memories are exemplaryrecording media for storing computer programs comprisingcomputer-executable instructions performing all the steps of thecomputer-implemented method according the technical concept presentedherein. The computer(s) can be placed within the operating room oroutside the operating room. Communication between the computers and thecomponents of the system may be performed by wire or wirelessly,according to known communication means.

The aim of the system is to generate, via the 3D display system 140, anaugmented reality image such as shown in examples of FIGS. 3F-3I andalso possibly 3A-3E. When the surgeon looks via the 3D display system140, the surgeon sees the augmented reality image 141A which comprises:

-   -   the real world image: the patient anatomy, surgeon's hands and        the instrument currently in use (which may be partially inserted        into the patient's body and hidden under the skin);    -   and a computer-generated surgical navigation image 142A        comprising the patient anatomy 163 configurable such that at        least one section of the anatomy 163A-163F is displayed and at        least one other section of the anatomy 163A-163F is not        displayed.

Furthermore, the surgical navigation image may further comprise a 3Dimage 171 representing at least one of: the virtual image of theinstrument 164 or surgical guidance indicating suggested (ideal)trajectory and placement of surgical instruments 107, according to thepre-operative plans 161 (as shown in FIG. 3C); preferably, threedifferent orthogonal planes of the patient anatomy data 163: coronal174, sagittal 173, axial 172; preferably, a menu 175 for controlling thesystem operation.

If the 3D display 142 is stereoscopic, the surgeon shall use a pair of3D glasses 151 to view the augmented reality image 141A. However, if the3D display 142 is autostereoscopic, it may be not necessary for thesurgeon to use the 3D glasses 151 to view the augmented reality image141A.

The virtual image of the patient anatomy 163 is generated based on datarepresenting a three-dimensional segmented model comprising at least twosections representing parts of the anatomy. The anatomy can be forexample a bone structure, such as a spine, skull, pelvis, long bones,shoulder joint, hip joint, knee joint etc. This description presentsexamples related particularly to a spine, but a skilled person willrealize how to adapt the embodiments to be applicable to the other bonystructures or other anatomy parts as well.

For example, the model can represent a spine, as shown in FIG. 6, withthe following section: spinous process 163A, lamina 163B, articularprocess 163C, transverse process 163D, pedicles 163E, vertebral body163F.

The model can be generated based on a pre-operative scan of the patientand then segmented manually by a user or automatically by a computer,using dedicated algorithms and/or neural networks, or in a hybridapproach including a computer-assisted manual segmentation. For example,a convolutional neural network such as explained with reference to FIGS.7-14 can be employed.

Preferably, the images of the orthogonal planes 172, 173, 174 aredisplayed in an area next (preferably, above) to the area of the 3Dimage 171, as shown in FIG. 3A, wherein the 3D image 171 occupies morethan 50% of the area of the see-through visor 141.

The location of the images of the orthogonal planes 172, 173, 174 may beadjusted in real time depending on the location of the 3D image 171,when the surgeon changes the position of the head during operation, suchas not to interfere with the 3D image 171.

Therefore, in general, the anatomical information of the user is shownin two different layouts that merge for an augmented and mixed realityfeature. The first layout is the anatomical information that isprojected in 3D in the surgical field. The second layout is in theorthogonal planes.

The surgical navigation image 142A is generated by the image generator131 in accordance with the tracking data provided by the fiducial markertracker 125, in order to superimpose the anatomy images and theinstrument images exactly over the real objects, in accordance with theposition and orientation of the surgeon's head. The markers are trackedin real time and the image is generated in real time. Therefore, thesurgical navigation image generator 131 provides graphics rendering ofthe virtual objects (patient anatomy, surgical plan and instruments)collocated to the real objects according to the perspective of thesurgeon's perspective.

For example, surgical guidance may relate to suggestions (virtualguidance clues 164) for placement of a pedicle screw in spine surgery orthe ideal orientation of an acetabular component in hip arthroplastysurgery. These suggestions may take a form of animations that show thesurgeon whether the placement is correct. The suggestions may bedisplayed both on the 3D holographic display and the orthogonal planes.The surgeon may use the system to plan these orientations before orduring the surgical procedure.

In particular, the 3D image 171 is adapted in real time to the positionand orientation of the surgeon's head. The display of the differentorthogonal planes 172, 173, 174 may be adapted according to the currentposition and orientation of the surgical instruments used.

FIG. 3B shows an example indicating collocation of the virtual image ofthe patient anatomy 163 and the real anatomy 105.

For example, as shown in FIG. 3C, the 3D image 171 may demonstrate amismatch between a supposed/suggested position of the instrumentaccording to the pre-operative plan 161, displayed as a first virtualimage of the instrument 164A located at its supposed/suggested position,and an actual position of the instrument, visible either as the realinstrument via the see-through display and/or a second virtual image ofthe instrument 164B overlaid on the current position of the instrument.Additionally, graphical guiding cues, such as arrows 165 indicating thedirection of the supposed change of position, can be displayed.

FIG. 3D shows a situation wherein the tip of the supposed position ofthe instrument displayed as the first virtual image 164A according tothe pre-operative plan 161 matches the tip of the real surgicalinstrument visible or displayed as the second virtual image 164B.However, the remaining objects do not match, therefore the graphicalcues 165 still indicate the need to change position. The surgicalinstrument is close to the correct position and the system may provideinformation on how close the surgical instrument is to the plannedposition.

FIG. 3E shows a situation wherein the supposed position of the realsurgical instrument matches the position of the instrument according tothe pre-operative plan 161, i.e. the correct position for surgery. Inthis situation the graphical cues 165 are no longer displayed, but thevirtual images 164A, 164B may be changed to indicate the correctposition, e.g. by highlighting it or blinking.

In some situations, the image of the full patient anatomy 163, as shownin FIG. 3A, may be obstructive. To solve this problem, the system allowsa selective display of the parts of the anatomy 163, such that at leastone part of the anatomy is shown and at least one other part of theanatomy is not shown.

For example, the surgeon may only want to see isolated parts of thespinal anatomy during spine surgery (only vertebral body or only thepedicle). Each part of the spinal anatomy is displayed at the request ofthe surgeon. For example the surgeon may only want to see the virtualrepresentation of the pedicle during placement of bony anchors. Thiswould be advantageous, as it would not have any visual interference fromthe surrounding anatomical structures.

Therefore, a single part of the anatomy may be displayed, for exampleonly the vertebral body 163F (FIG. 3F) or only the pedicles 163E (FIG.3G). Alternatively, two parts of the anatomy may be displayed, forexample the vertebral body 163F and the pedicles 163E (FIG. 3H); or alarger group of anatomy parts may be displayed, such as the top parts of163A-D of the spine (FIG. 3I).

The user may select the parts that are to be displayed via the inputinterface 132.

For example, the GUI may comprise a set of predefined display templates,each template defining a particular part of the anatomy to be displayed(such as FIG. 3F, 3G) or a plurality of parts of the anatomy to bedisplayed (such as FIG. 3H, 3I). The user may then use a dedicatedtouch-screen button, keyboard key, pedal or other user interfacenavigation element to select a particular template to be displayed or toswitch between consecutive templates.

Alternatively, the GUI may display a list of available parts of anatomyto be displayed and the user may select the parts to be displayed.

The GUI interface for configuring the parts that are to be displayed canbe configured to be operated directly by the surgeon or by an assistantperson.

The foregoing description will provide examples of a 3D display 142 witha see-through mirror 141, which is particularly effective to provide thesurgical navigation data. However, other 3D display systems can be usedas well to show the automatically segmented parts of anatomy, such as 3Dhead-mounted displays.

The see-through mirror (also called a half-silvered mirror) 141 is atleast partially transparent and partially reflective, such that theviewer can see the real world behind the mirror but the mirror alsoreflects the surgical navigation image generated by the displayapparatus located above it.

For example, a see-through mirror as commonly used in teleprompters canbe used. For example, the see-through mirror 141 can have a reflectiveand transparent rate of 50R/50T, but other rates can be used as well.

The surgical navigation image is emitted from above the see-throughmirror 141 by the 3D display 142.

In an example embodiment as shown in FIGS. 4A and 4B, a special designof the 3D display 142 is provided that is compact in size to facilitateits mounting within a limited space at the operating room. That designallows generating images of relatively large size, taking into accountthe small distance between the 3D display 142 and the see-through mirror141, without the need to use wide-angle lens that could distort theimage.

The 3D display 142 comprises a 3D projector 143, such as a DLPprojector, that is configured to generate an image, as shown in FIG. 4B(by the dashed lines showing image projection and solid lines showingimages generated on particular reflective planes). The image from the 3Dprojector 143 is firstly refracted by an opaque top mirror 144, then itis refracted by an opaque vertical mirror 145 and subsequently placed onthe correct dimensions on a projection screen 146 (which can be simply aglass panel). The projection screen 146 works as a rear-projectionscreen or a small bright 3D display. The image displayed at theprojection screen 146 is reflected by the see-through mirror 141 whichworks as an augmented reality visor. Such configuration of the mirrors144, 145 allows the image generated by the 3D projector 143 to be shownwith an appropriate size at the projection screen 146. The fact that theprojection screen 146 emits an enlarged image generated by the 3Dprojector 143 makes the emitted surgical navigation image bright, andtherefore well visible when reflected at the see-through mirror 141.Reference 141A indicates the augmented reality image as perceived by thesurgeon when looking at the see-through mirror 141.

The see-through mirror 141 is held at a predefined position with respectto the 3D projector 143, in particular with respect to the 3D projector143, by an arm 147, which may have a first portion 147A fixed to thecasing of the 3D display 142 and a second portion 147B detachably fixedto the first portion 147A. The first portion 147A may have a protectivesleeve overlaid on it. The second portion 147B, together with thesee-through mirror 141, may be disposable in order to keep sterility ofthe operating room, as it is relatively close to the operating field andmay be contaminated during the operation. The arm can also be foldableupwards to leave free space of the work space when the arm and augmentedreality are not needed.

In alternative embodiments, as shown for example in FIGS. 4C, 4D, 4E,alternative devices may be used in the 3D display system 140 in place ofthe see-through mirror 141 and the 3D display 142.

As shown in FIG. 4C, a 3D monitor 146A can be used directly in place ofthe projection screen 146.

As shown in FIG. 4D, a 3D projector 143 can be used instead of the 3Ddisplay 142 of FIG. 4A, to project the surgical navigation image onto asee-through projection screen 141B, which is partially transparent andpartially reflective, for showing the surgical navigation image 142A andallowing the surgeon to see the surgical field 108. A lens 141C can beused to provide appropriate focal position of the surgical navigationimage.

As shown in FIG. 4E, the surgical navigation image can be displayed at athree-dimensional see-through screen 141D and viewed by the user via alens 141C used to provide appropriate focal position of the surgicalnavigation image.

Therefore, see-through screen 141B, the see-through display 141D and thesee-through mirror 141 can be commonly called a see-through visor.

If a need arises to adapt the position of the augmented reality screenwith respect to the surgeon's head (for example, to accommodate theposition depending on the height of the particular surgeon), theposition of the whole 3D display system 140 can be changed, for exampleby manipulating an adjustable holder (a surgical boom) 149 on FIG. 1A,by which the 3D display 142 is attachable to an operating roomstructure, such as a ceiling, a wall or a floor.

An eye tracker 148 module can be installed at the casing of the 3Ddisplay 142 or at the see-through visor 141 or at the wearable glasses151, to track the position and orientation of the eyes of the surgeonand input that as commands via the gaze input interface to control thedisplay parameters at the surgical navigation image generator 131, forexample to activate different functions based on the location that isbeing looked at, as shown in FIGS. 5A and 5B.

For example, the eye tracker 148 may use infrared light to illuminatethe eyes of the user without affecting the visibility of the user,wherein the reflection and refraction of the patterns on the eyes areutilized to determine the gaze vector (i.e. the direction at which theeye is pointing out). The gaze vector along with the position andorientation of the user's head is used to interact with the graphicaluser interface. However, other eye tracking algorithms techniques can beused as well.

It is particularly useful to use the eye tracker 148 along with thepedals 132 as the input interface, wherein the surgeon may navigate thesystem by moving a cursor by eyesight and inputting commands (such asselect or cancel) by pedals.

FIGS. 7-14 show an example of a convolutional neural network (CNN) thatcan be used to automatically segment the bone structure to provideanatomy section data for the selective display as described above.

The CNN can be used to process images of a bony structure, such as aspine, skull, pelvis, long bones, shoulder joint, hip joint, knee jointetc. The foregoing description will present examples related mostly to aspine, but a skilled person will realize how to adapt the embodiments tobe applicable to the other bony structures as well.

Moreover, the CNN may include, before segmentation, pre-processing oflower quality images to improve their quality. For example, the lowerquality images may be low dose computed tomography (LDCT) images ormagnetic resonance images captured with a relatively low power scannercan be denoised. The foregoing description will present examples relatedto computed tomography (CT) images, but a skilled person will realizehow to adapt the embodiments to be applicable to other image types, suchas magnetic resonance images.

FIGS. 7A-7E show examples of various CT images of a spine. FIGS. 7F-7Jshow their corresponding segmented images obtained by the methodpresented herein.

FIGS. 8A and 8B show an enlarged view of a CT scan, wherein FIG. 8A isan image with a high noise level (such as a low dose (LDCT) image) andFIG. 8B is an image with a low noise level (such as a high dose (HDCT)image or a LDCT image denoised according to the method presentedherein).

FIG. 8C shows a low strength magnetic resonance scan of a neck portionand FIG. 8D shows a higher strength magnetic resonance scan of the sameneck portion (wherein FIG. 8D is also the type of image that is expectedto be obtained by performing denoising of the image of FIG. 8C).

Therefore, in the present CNN, a low-dose medical imagery (such as shownin FIG. 8A, 8C) is pre-processed to improve its quality to the qualitylevel of a high-dose or high quality medical imagery (such as shown inFIG. 8B, 8D), without the need to expose the patient to the high doseimagery.

For the purposes of this disclosure, the LDCT image is understood as animage which is taken with an effective dose of X-ray radiation lowerthan the effective dose for the HDCT image, such that the lower dose ofX-ray radiation causes appearance of higher amount of noise on the LDCTimage than the HDCT image. LDCT images are commonly captured duringintra-operative scans to limit the exposure of the patient to X-rayradiation.

As seen by comparing FIGS. 8A and 8B, the LDCT image is quite noisy andis difficult to be automatically processed by a computer to identify thecomponents of the anatomical structure.

The system and method disclosed below use a neural network anddeep-learning based approach. In order for any neural network to work,it must first be learned. The learning process is supervised (i.e., thenetwork is provided with a set of input samples and a set ofcorresponding desired output samples). The network learns the relationsthat enable it to extract the output sample from the input sample. Givenenough training examples, the expected results can be obtained.

In the presented system and method, a set of samples are generatedfirst, wherein LDCT images and HDCT images of the same object (such asan artificial phantom or a lumbar spine) are captured using the computedtomography device. Next, the LDCT images are used as input and theircorresponding HDCT images are used as desired output to learn theneutral network to denoise the images. Since the CT scanner noise is nottotally random (there are some components that are characteristic forcertain devices or types of scanners), the network learns which noisecomponent is added to the LDCT images, recognizes it as noise and it isable to eliminate it in the following operation, when a new LDCT imageis provided as an input to the network.

By denoising the LDCT images, the presented system and method may beused for intra-operative tasks, to provide high segmentation quality forimages obtained from intra-operative scanners on low radiation dosesetting.

FIG. 10 shows a convolutional neural network (CNN) architecture 300,hereinafter called the denoising CNN, which is utilized in the presentmethod for denoising. The network comprises convolution layers 301 (withReLU activation attached) and deconvolution layers 302 (with ReLUactivation attached). The use of a neural network in place of standardde-noising techniques provides improved noise removal capabilities.Moreover, since machine learning is involved, the network can be tunedto specific noise characteristics of the imaging device to furtherimprove the performance. This is done during training. The architectureis general, in the sense that adopting it to images of different size ispossible by adjusting the size (resolution) of the layers. The number oflayers and the number of filters within layers is also subject tochange, depending on the requirements of the application. Deepernetworks with more filters typically give results of better quality.However, there's a point at which increasing the number oflayers/filters does not result in significant improvement, butsignificantly increases the computation time, making such a largenetwork impractical.

FIG. 11 shows a convolutional neural network (CNN) architecture 400,hereinafter called the segmentation CNN, which is utilized in thepresent method for segmentation (both semantic and binary). The networkperforms pixel-wise class assignment using an encoder-decoderarchitecture, using as input the raw images or the images denoised withthe denoising CNN. The left side of the network is a contracting path,which includes convolution layers 401 and pooling layers 402, and theright side is an expanding path, which includes upsampling or transposeconvolution layers 403 and convolutional layers 404 and the output layer405.

One or more images can be presented to the input layer of the network tolearn reasoning from single slice image, or from a series of imagesfused to form a local volume representation.

The convolution layers 401 can be of a standard kind, the dilated kind,or a combination thereof, with ReLU or leaky ReLU activation attached.

The upsampling or deconvolution layers 403 can be of a standard kind,the dilated kind, or a combination thereof, with ReLU or leaky ReLUactivation attached.

The output slice 405 denotes the densely connected layer with one ormore hidden layer and a softmax or sigmoid stage connected as theoutput.

The encoding-decoding flow is supplemented with additional skippingconnections of layers with corresponding sizes (resolutions), whichimproves performance through information merging. It enables either theuse of max-pooling indices from the corresponding encoder stage todownsample, or learning the deconvolution filters to upsample.

The architecture is general, in the sense that adopting it to images ofdifferent size is possible by adjusting the size (resolution) of thelayers. The number of layers and number of filters within a layer isalso subject to change, depending on the requirements of theapplication.

Deeper networks typically give results of better quality. However, thereis a point at which increasing the number of layers/filters does notresult in significant improvement, but significantly increases thecomputation time and decreases the network's capability to generalize,making such a large network impractical.

The final layer for binary segmentation recognizes two classes (bone andno-bone). The semantic segmentation is capable of recognizing multipleclasses, each representing a part of the anatomy. For example, for thevertebra, this includes vertebral body, pedicles, processes etc.

FIG. 11 shows a flowchart of a training process, which can be used totrain both the denoising CNN 300 and the segmentation CNN 400.

The objective of the training for the denoising CNN 300 is to tune theparameters of the denoising CNN 300 such that the network is able toreduce noise in a high noise image, such as shown in FIG. 8A, to obtaina reduced noise image, such as shown in FIG. 8B.

The objective of the training for the segmentation CNN 400 is to tunethe parameters of the segmentation CNN 400 such that the network is ableto recognize segments in a denoised image (such as shown in FIGS. 7A-7Eor FIG. 8A) to obtain a segmented image (such as shown in FIGS. 7F-7J orFIG. 8B), wherein a plurality of such segmented images can be thencombined to a 3D segmented image such as shown in FIG. 6.

The training database may be split into a training set used to train themodel, a validation set used to quantify the quality of the model, and atest set.

The training starts at 501. At 502, batches of training images are readfrom the training set, one batch at a time. For the denoising CNN, LDCTimages represent input, and HDCT images represent desired output. Forthe segmentation CNN, denoised images represent input, and pre-segmented(by a human) images represent output.

At 503 the images can be augmented. Data augmentation is performed onthese images to make the training set more diverse. The input/outputimage pair is subjected to the same combination of transformations fromthe following set: rotation, scaling, movement, horizontal flip,additive noise of Gaussian and/or Poisson distribution and Gaussianblur, etc.

At 504, the images and generated augmented images are then passedthrough the layers of the CNN in a standard forward pass. The forwardpass returns the results, which are then used to calculate at 505 thevalue of the loss function—the difference between the desired output andthe actual, computed output. The difference can be expressed using asimilarity metric, e.g.: mean squared error, mean average error,categorical cross-entropy or another metric.

At 506, weights are updated as per the specified optimizer and optimizerlearning rate. The loss may be calculated using a per-pixelcross-entropy loss function and the Adam update rule.

The loss is also back-propagated through the network, and the gradientsare computed. Based on the gradient values, the network's weights areupdated. The process (beginning with the image batch read) is repeatedcontinuously until an end of the training session is reached at 507.

Then, at 508, the performance metrics are calculated using a validationdataset—which is not explicitly used in training set. This is done inorder to check at 509 whether not the model has improved. If it isn'tthe case, the early stop counter is incremented at 514 and it is checkedat 515 if its value has reached a predefined number of epochs. If so,then the training process is complete at 516, since the model hasn'timproved for many sessions now.

If the model has improved, the model is saved at 510 for further use andthe early stop counter is reset at 511. As the final step in a session,learning rate scheduling can be applied. The session at which the rateis to be changed are predefined. Once one of the session numbers isreached at 512, the learning rate is set to one associated with thisspecific session number at 513.

Once the training is complete, the network can be used for inference,i.e. utilizing a trained model for prediction on new data.

FIG. 12 shows a flowchart of an inference process for the denoising CNN300.

After inference is invoked at 601, a set of scans (LDCT, not denoised)are loaded at 602 and the denoising CNN 300 and its weights are loadedat 603.

At 604, one batch of images at a time is processed by the inferenceserver. At 605, a forward pass through the denoising CNN 300 iscomputed.

At 606, if not all batches have been processed, a new batch is added tothe processing pipeline until inference has been performed at all inputnoisy LDCT images.

Finally, at 607, the denoised scans are saved.

FIG. 13 shows a flowchart of an inference process for the segmentationCNN 400.

After inference is invoked at 701, a set of scans (denoised imagesobtained from noisy LDCT images) are loaded at 702 and the segmentationCNN 400 and its weights are loaded at 703.

At 704, one batch of images at a time is processed by the inferenceserver.

At 705, the images are preprocessed (e.g., normalized, cropped) usingthe same parameters that were utilized during training, as discussedabove. In at least some implementations, inference-time distortions areapplied and the average inference result is taken on, for example, 10distorted copies of each input image. This feature creates inferenceresults that are robust to small variations in brightness, contrast,orientation, etc.

At 706, a forward pass through the segmentation CNN 400 is computed.

At 707, the system may perform post-processing such as linear filtering(e.g. Gaussian filtering), or nonlinear filtering, such as medianfiltering and morphological opening or closing.

At 708, if not all batches have been processed, a new batch is added tothe processing pipeline until inference has been performed at all inputimages.

Finally, at 709, the inference results are saved and can be combined toa segmented 3D model. The model can be further converted to a polygonalmesh representation for the purpose of visualization on the display. Thevolume and/or mesh representation parameters can be adjusted in terms ofchange of color, opacity, changing the mesh decimation depending on theneeds of the operator.

FIG. 14A shows a sample image of a CT spine scan and FIG. 14B shows asample image of its segmentation. Every class (anatomical part of thevertebrae) can be denoted with its specific color. The segmented imagecomprises spinous process 11, lamina 12, articular process 13,transverse process 14, pedicles 15, vertebral body 16.

FIG. 6 shows a sample of the segmented images displaying all the partsof the vertebrae (11-16) obtained after the semantic segmentationcombined into a 3D model.

The functionality described herein can be implemented in a computersystem. The system may include at least one non-transitoryprocessor-readable storage medium that stores at least one ofprocessor-executable instructions or data and at least one processorcommunicably coupled to that at least one non-transitoryprocessor-readable storage medium. That at least one processor isconfigured to perform the steps of the methods presented herein.

FIG. 15 shows a schematic illustration of a computer-implemented system900, for example a machine learning system, in accordance with oneembodiment of the invention, for implementing the segmentation CNN. Thesystem 900 may include at least one non-transitory processor-readablestorage medium 910 that stores at least one of processor-executableinstructions 915 or data; and at least one processor 920 communicablycoupled to the at least one non-transitory processor-readable storagemedium 910. The at least one processor 920 may be configured to (byexecuting the instructions 915) receive segmentation learning datacomprising a plurality of batches of labeled anatomical image sets, eachimage set comprising image data representative of a series of slices ofa three-dimensional bony structure, and each image set including atleast one label which identifies the region of a particular part of thebony structure depicted in each image of the image set, wherein thelabel indicates one of a plurality of classes indicating parts of thebone anatomy. The at least one processor 920 may also be configured to(by executing the instructions 915) train a segmentation CNN, that is afully convolutional neural network model with layer skip connections, tosegment into plurality of classes at least one part of the bonystructure utilizing the received segmentation learning data. The atleast one processor 920 may also be configured to (by executing theinstructions 915) store the trained segmentation CNN in at least onenon-transitory processor-readable storage medium 910 of the machinelearning system.

While the invention has been described with respect to a limited numberof embodiments, it will be appreciated that many variations,modifications and other applications of the invention may be made.Therefore, the claimed invention as recited in the claims that follow isnot limited to the embodiments described herein.

What is claimed is:
 1. A surgical navigation system comprising: a sourceof a patient anatomy data; wherein the patient anatomy data comprises athree-dimensional reconstruction of a segmented model comprising atleast two sections representing parts of the anatomy; a surgicalnavigation image generator configured to generate a surgical navigationimage comprising the patient anatomy; a 3D display system configured toshow the surgical navigation image wherein the display of the patientanatomy is selectively configurable such that at least one section ofthe anatomy is displayed and at least one other section of the anatomyis not displayed.
 2. The system of claim 1, further comprising: atracking system for real-time tracking of a surgeon's head, asee-through visor of the 3D display system and a patient anatomy toprovide current position and/or orientation data; wherein the surgicalnavigation image generator is configured to generate the surgicalnavigation image in accordance to the current position and/ororientation data provided by the tracking system.
 3. The system of claim1, further comprising: a source of at least one of: an operative plan(161, 162) and a virtual surgical instrument model; wherein the trackingsystem is further configured for real-time tracking of surgicalinstruments; wherein the surgical navigation image further comprises athree-dimensional image representing a virtual image of the surgicalinstruments.
 4. The system of claim 3, wherein the virtual image of thesurgical instruments is configured to indicate the suggested positionsand/or orientations of the surgical instruments according to theoperative plan data.
 5. The system of claim 4, wherein thethree-dimensional image of the surgical navigation image furthercomprises a graphical cue indicating the required change of positionand/or orientation of the surgical instrument to match the suggestedposition and/or orientation according to the preoperative plan data. 6.The system of claim 1, wherein the surgical navigation image furthercomprises a set of orthogonal (axial, sagittal, and coronal) and/orarbitrary planes of the patient anatomy data.
 7. The system of claim 2,wherein the 3D display system is configured to show the surgicalnavigation image at a see-through visor, such that an augmented realityimage collocated with the patient anatomy in the surgical fieldunderneath the see-through visor is visible to a viewer looking fromabove the see-through visor towards the surgical field.
 8. The system ofclaim 1, wherein the patient anatomy data comprises output data of asemantic segmentation process of an anatomy scan image.
 9. The system ofclaim 8, further comprising a convolutional neural network (CNN) systemconfigured to perform the semantic segmentation process to generate thepatient anatomy data.
 10. The system of claim 9, wherein theconvolutional neural network (CNN) system comprises: at least onenon-transitory processor-readable storage medium that stores at leastone of processor-executable instructions or data; and at least oneprocessor communicably coupled to at least one non-transitoryprocessor-readable storage medium, wherein that at least one processor:receives segmentation learning data comprising a plurality of batches oflabeled anatomical image sets, each image set comprising image datarepresentative of a series of slices of a three-dimensional bonystructure of the anatomy, and each image set including at least onelabel which identifies the region of a particular part of the bonystructure depicted in each image of the image set, wherein the labelindicates one of a plurality of classes indicating parts of the boneanatomy; trains a segmentation CNN, that is a fully convolutional neuralnetwork model with layer skip connections) to segment semantically atleast one part of the bony structure utilizing the received segmentationlearning data; and stores the trained segmentation CNN in at least onenon-transitory processor-readable storage medium of the machine learningsystem.
 11. The system of claim 1, wherein at least one processorfurther: receives denoising learning data comprising a plurality ofbatches of high quality medical images and low quality medical images,wherein the high quality medical images have a lower noise level thanthe low quality medical images; trains a denoising CNN, that is a fullyconvolutional neural network model with layer skip connections todenoise an image utilizing the received denoising learning data; andstores the trained denoising CNN in at least one non-transitoryprocessor-readable storage medium of the machine learning system. 12.The system of claim 11, wherein at least one processor further operatesthe trained segmentation CNN to process a set of input anatomical imagesto generate a set of output segmented anatomical images.
 13. The systemof claim 11, wherein at least one processor further operates the traineddenoising CNN to process a set of input anatomical images to generate aset of output denoised anatomical images.
 14. The system of claim 13,wherein the set of input anatomical images for the trained denoising CNNcomprises the low quality anatomical images.
 15. A method for providingan augmented reality image during an operation, comprising: providing asource of a patient anatomy data; wherein the patient anatomy datacomprises a three-dimensional reconstruction of a segmented modelcomprising at least two sections representing parts of the anatomy;generating, by a surgical navigation image generator, a surgicalnavigation image comprising the patient anatomy; showing the surgicalnavigation image at 3D display system and selectively configuring thedisplay of the patient anatomy such that at least one section of theanatomy is displayed and at least one other section of the anatomy isnot displayed.